Deep learning acceleration of multiscale superresolution localization photoacoustic imaging

A superresolution imaging approach that localizes very small targets, such as red blood cells or droplets of injected photoacoustic dye, has significantly improved spatial resolution in various biological and medical imaging modalities. However, this superior spatial resolution is achieved by sacrificing temporal resolution because many raw image frames, each containing the localization target, must be superimposed to form a sufficiently sampled high-density superresolution image. Here, we demonstrate a computational strategy based on deep neural networks (DNNs) to reconstruct high-density superresolution images from far fewer raw image frames. The localization strategy can be applied for both 3D label-free localization optical-resolution photoacoustic microscopy (OR-PAM) and 2D labeled localization photoacoustic computed tomography (PACT). For the former, the required number of raw volumetric frames is reduced from tens to fewer than ten. For the latter, the required number of raw 2D frames is reduced by 12 fold. Therefore, our proposed method has simultaneously improved temporal (via the DNN) and spatial (via the localization method) resolutions in both label-free microscopy and labeled tomography. Deep-learning powered localization PA imaging can potentially provide a practical tool in preclinical and clinical studies requiring fast temporal and fine spatial resolutions.


Supplementary Text Supplementary Materials and Methods
. Label-free localization-based OR-PAM imaging process. Fig. S2. Labeled localization-based PACT imaging process. Fig. S3. Customized 2D and 3D discriminator network architecture. Fig. S4. 3D Graphs of evaluation metrics depending on frame/droplet counts used to reconstruct a sparse localization-based image for the training and test set. Fig. S5. Configuration of an optical-resolution photoacoustic microscopy system. Fig. S6. Spatial resolution of an optical-resolution photoacoustic microscopy system. Fig. S7. Configuration of a photoacoustic computed tomography system. Table S1. Summary of generator networks. Table S2. Details of the discriminator networks. Table S3. Comparison of 3D PSNR and 3D MS-SSIM metrics depending on frame counts used to reconstruct a sparse localization-based image for the training and test set. Table S4. Comparison of 2D PSNR and 2D MS-SSIM metrics depending on droplet counts used to reconstruct a sparse localization-based image for the training and test sets. Table S5. Hyper parameters for training.

Movie files:
Movie S1. Formation of sparse, DNN, and dense localization-based OR-PAM images. Movie S2. Formation of sparse, DNN, and dense localization-based PACT images.

Label-free localization optical-resolution photoacoustic microscopy (OR-PAM)
We adopted a previously-reported localization processing procedure to reconstruct label-free super-resolution OR-PAM images (Fig. S1) 1 . The localization processing was applied to OR-PAM volumetric images obtained by a galvanometer scanner OR-PAM (OptichoM, Opticho, South Korea). The system is described in Fig. S5. The OR-PAM system, with a fast temporal resolution (B-scan speed of 500 Hz), could capture intrinsic red blood cells (RBCs) instantaneously, generating photoacoustic (PA) signals from their locations (Fig. S1). Because each frame captured the signals in the flowing blood at different points, a densely-connected localization image could be reconstructed from multiple OR-PAM frames obtained while continuously imaging the same region of a mouse ear. Finally, localization frames were translated from OR-PAM frames by localizing the PA signals within the frames, and then were superimposed to create a dense localization-based OR-PAM image.

Labeled localization photoacoustic computed tomography (PACT)
A previously-reported localization algorithm for PACT was applied to produce datasets for a 2D deep neural network (Fig. S2) 2 . A PACT system (Fig. S7) continuously imaged the cortical layer of a mouse brain during injection of small dyed droplets that had a higher optical absorption contrast than RBCs at the optical wavelength of 780 nm. Thanks to the higher contrast, the droplets could be tracked and localized with high precision. Imaging for half an hour at a frame rate of 20 Hz resulted in a total of 36,000 frames. As in the localization OR-PAM environment, the flow within the blood vessel caused each droplet to be detected at different subsequent positions. In the localization PACT processing, droplets were extracted from the acquired PACT images, and then all the localized droplets were combined to produce a dense localization-based PACT image (Fig.  S2).

Supplementary Materials and Methods:
Label-free localization OR-PAM The regular OR-PAM images for training were obtained using an OR-PAM system ( Fig. S5) with an optical fiber to deliver optical pulses onto the target. Thanks to the fiber, we could obtain volumetric images from a fixed target, reducing motion artifacts during the imaging experiments. To excite a target, the system uses a fast nano-pulse laser system with a maximum pulse repetition rate of 600 kHz (VPFL-G-10, Spectra-Physics, USA). The laser beams are collimated with a fiber optic collimator (TC25FC-543, Thorlabs, USA) and then focused by an objective lens (LA1213-A, Thorlabs, USA) (Figs. S5a, b). The laser beams pass through the center hole of a customized ring-shaped ultrasound transducer with a focal length of 21 mm, an outer diameter of 15 mm, an inner diameter of 2.5 mm, a central frequency of 20 MHz, and a bandwidth of 60%. In this PAM system, we used a galvanometer scanner (GVS001, Thorlabs, USA), and we newly designed the mirror of the galvanometer scanner to steer optical beams downward (Fig. S5b). The focused beam is reflected by the mirror, steered by the scanner, and irradiates the target, thereby inducing PA waves. The PA waves are then reflected by the mirror of the scanner and measured by the transducer. A multifunctional data acquisition board (DAQ, NI PCIe-6321, National Instruments, USA) synchronizes all the mechanical systems (i.e., the laser system, galvanometer scanner, linear motorized stages, and the digitizer). B-scan images are obtained by fast angular scanning of the galvanometer scanner, and volumetric images are acquired by scanning of the linear motorized stage slowly during the fast angular scanning. The maximum B-mode imaging speed reached 500 Hz, with lateral and axial resolutions of 9.1 μm and of 114 μm (Fig. S6), respectively, under a scanning range of ~1.5 mm, 400 pixels, and a laser repetition rate of 400 kHz. The measured resolutions matched well with the theoretical lateral and axial resolutions of 8.5 μm and 113 μm, respectively, for an optical numerical aperture of 0.032, a central frequency of 20 MHz, and a -6dB bandwidth of 60% 3 . The measured PA signals, pre-amplified by an amplifier (ZX60-3018G-S+, Mini-Circuits, 26-dB gain, USA), are finally transferred into digital signals by the digitizer (ATS-9350, Alarzatech, USA) and saved in binary format.
Localization-based OR-PAM images were reconstructed through the following processes: Fast-acquired volumetric OR-PAM images were precisely aligned via an intensity-based image registration algorithm 4 . The aligned volume data was spatially interpolated to a size of 4x along xand y-axes by bicubic interpolation. After being normalized and convolved with an averaging filter with a kernel size of 3 × 3 × 3 to emphasize local maximum points, the OR-PAM images were transferred into volumetric localization frames by determining the local maximum points in MATLAB. A super resolution volumetric localization OR-PAM image was then reconstructed by superimposing all the localization frames. The localization OR-PAM imaging improved the spatial resolution by a factor of 2.5 in vivo 1 .

Labeled localization PACT
The PACT system used in this study is shown in Fig. S7. A Ti: Sapphire laser (LS-2145-LT-150, Symphotic Tii; 20 Hz pulse repetition rate; 12 ns pulse width) is used to output 780 nm pulses for PA excitation. The laser beam is first homogenized by an optical diffuser (EDC-5, RPC Photonics) and then illuminates the mouse brain from above. The PA signals are detected by a full-ring ultrasonic transducer array (Imasonic) with a 10-cm diameter, a 5-MHz central frequency, more than 90% one-way bandwidth, and 512 elements. Each element (20-mm height, 0.61-mm pitch, and 0.1-mm inter-element space) is cylindrically focused to produce an axial focal distance of 45 mm (acoustic NA, 0.2). The combined foci of all 512 elements form an approximately uniform imaging region with a 20-mm diameter and 1-mm thickness. In our experiments, a lab-made 512channel preamplifier (26 dB gain) was directly connected to the ultrasonic transducer array housing, with minimized connection cable length to minimize cable noise. The pre-amplified photoacoustic signals were digitized using a 512-channel data acquisition system (four SonixDAQs, Ultrasonix Medical ULC; 128 channels each; 40 MHz sampling rate; 12 bits dynamic range) with programmable amplification up to 51 dB. The digitized radio frequency data were first stored in the onboard buffer, then transferred to a computer. The digitized raw data were fed into a half-time dual-speed-of-sound universal back-projection algorithm for image reconstruction 5 . The in-plane resolution of this system was previously quantified as ~150 µm for an imaging size of 10 mm × 12 mm and a pixel size of 25 μm 6 . In the PACT localization experiments, IR-780, an iodide hydrophobic dye (425311, Sigma-Aldrich), was used as an optical contrast agent in the droplets. A mixture of 67% (v/v) clove oil (C8392, Sigma-Aldrich) and 33% (v/v) peanut oil (P2144, Sigma-Aldrich) was the solvent. The oil mixture was prepared so that the final solution had a density close to that of water, which guaranteed good stability of the droplets in both water and whole blood. It took 24-48 hours to fully dissolve the dye in the oil solvent, obtaining a maximum concentration of 2 mM. A mixture of 20 μL of the dye solution (2 mM) and 2 μL of surfactant (span® 80, S6760-250ML, Sigma-Aldrich) was added to 1 mL of distilled water and then vibrated for 10 s to form a droplet suspension. The final droplet suspension had a concentration of ~ 4×10 7 mL -1 .
To reconstruct a localization PACT image, a cortical layer of a mouse brain was imaged for 30 minutes with a frame rate of 20 Hz during droplet injection (Fig. S7). To trace the droplets in the brain, the time-lapse PACT images were first denoised by applying a 2-D adaptive noiseremoval filter 7 , then subtraction of adjacent frames highlighted moving droplets. To localize the single droplets, the differential images were converted into binary images by thresholding the pixel values at 1/4 of their maxima. The bright spots within a range of 16 to 64 pixels in the binary images were the regions containing droplets. Any spots with a roundness of less than 0.7 were abandoned, which removed droplet clusters and artifacts. The centroids of the bright spots in the binary images were determined to coarsely locate single droplets in the differential images. Then, a ROI of each droplet, centered at its centroid, was isolated from the differential images. The ROI (11 × 11 pixels) was fitted with a 2-D Gaussian function, yielding a precise localization of the center of each droplet. Every droplet was characterized by a 2-D Gaussian-distributed spot with a radius equal to its localization uncertainty. Adding up all the droplets yielded a super-resolution localization PACT image, which improved the spatial resolution by a factor of 6 in vivo. :   Fig. S1. Label-free localization-based OR-PAM imaging process. A regular OR-PAM frame is first translated into a localization frame, and then all the localization frames are superimposed to reconstruct a dense localization-based OR-PAM image. PA, photoacoustic; OR-PAM, optical-resolution photoacoustic microscopy; RBC, red blood cell; Dense local., dense localization-based PA image.           30 38.12±2.69 38.62±2.29 39.08±2.04 41.37±2.12 41.22±2.08 41.32±2.21 41.27±2.15 40.64±2.21 39.62±2.12 10 39. 73±2.28 37.66±2.68 38.22±2.29 38.84±1.99 41.46±2.14 41.40±2.03 41.63±2.23 41.63±2.14 41.16±2.21